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BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention relates to a browsing of a multi-media data, such as a moving 
picture, and more particularly, to system, method, and multi-level object data structure thereof 
5 for browsing a multi -media data, which permits to browse detailed supplementary information 
on each object included in the multi-media data in displaying the multi-media data on request 
from a user. 

Background of the Related Art 

Recently, as searching techniques for digital moving picture are developed, there have 

10 been active suggestions on browsing or searching contents based moving picture. As typical 
examples, there are a browsing by using a key frame disclosed in the USP 5 ,708 ,767, or an obj ect 
based browsing method disclosed in the USP 5,655, 1 1 7. In the browsing by using a key frame, 
a few typical important scenes/key frames are suggested, and reproduced in an entire moving 
picture, to search the moving picture. However, since this method is conducted merely relying 

1 5 on the key frames, it is difficult to expect a search according to actual development of contents 
of the moving picture. On the other hand, in the object based moving picture searching, the 
moving picture is depicted and searched with reference to entrance and exit of different objects, 
such as characters, in the moving picture. However, most of the objects defined in the object 
based moving picture searching is foreground objects, of which structures are not appropriate for 

20 expressing place information which is background objects. Therefore, the background 
information, such as a place, can not be used in the related art, because, though a portion a 
specific object enters on the picture may be defined as an actual entry section in the case of 
foreground objects, the place is defined as one entry section since the place is the same as far as 



one scene is continued even if a camera angle is changed. Therefore, the place has implication 
different from a simple entry section in other foreground object as an implication that the places 
are the same is reflected to an object entry section even if appearances of an object on the picture 
are different. And, even though a foreground information with a motion is information only on 

5 movement of the object, place information with a motion implies a movement of the camera. 
Such place information becomes more useful when the place information is managed as a 
separate object. However, the watcher may want supplementary information on an actual place, 
i.e., where is the place the picture is taken, or other supplementary information on the actual 
place. For example, though a cafe where the heroine drops in is place information of the moving 

10 picture contents, the watcher may want supplementary information, such as the place the cafe 
actually is, or an introduction on an address or place(or a sketch map). Or the watcher may want 
detailed supplementary information on a particular object in the moving picture in view of 
contents of the moving picture, or actual supplementary information(for example, name, brief 
profile, and information on a home page, an address, or fan club, for fans). However, the related 

15 art method has a problem in that a searching or browsing method which can meet the 
aforementioned users wants can not be provided. 

SUMMARY OF THE INVENTION 
Accordingly, the present invention is directed to system, method, and multi-level object 
data structure thereof for browsing a multi-media data that substantially obviates one or more of 

20 the problems due to limitations and disadvantages of the related art. 

An object of the present invention is to provide system, and method for browsing a multi- 
media data, which permits to browse or search for supplementary information on a particular 
object included in a moving picture based on three elements of object, event, and background 



during the watcher watches the moving picture. 

Other object of the present invention is to provide a multi-level object data structure for 
implementing the system, and method for browsing a multi-media data of the present invention. 
Additional features and advantages of the invention will be set forth in the description 
5 which follows, and in part will be apparent from the description, or may be learned by practice 
of the invention. The objectives and other advantages of the invention will be realized and 
attained by the structure particularly pointed out in the written description and claims hereof as 
well as the appended drawings. 

To achieve these and other advantages and in accordance with the purpose of the present 
1 0 invention, as embodied and broadly described, the method for browsing a multimedia data using 
a multilevel obj ect data structure, including the steps of ( 1 ) receiving multimedia information and 
multiple supplementary information on each object in the multimedia information on the same 
time, (2) separating the multimedia information and the multiple supplementary information, (3) 
displaying the multimedia information, and (4) browsing/searching supplementary information 
15 related to a particular object from the multiple supplementary information, and displaying the 
supplementary information when a user requests for browsing/searching the supplementary 
information related to the particular object. 

The obj ect related supplementary information includes content information on a particular 
program in the multimedia information, and multiple supplementary information including real 
20 information. 

The content information and the real information includes selective combinations of 
information which represents relations of object information, event information, place 
information, and object/event/place information in a graph. 



The content information and the real information includes text information describing 
character of the object, and image or graphic information on the character. 

The object information includes text information having information describing the 
object, and image information on the object information. 
5 The event information includes text information describing contents of the event, and 

information on a place or a sketch map the event occurred. 

The place information includes text information describing the place, and information on 
location or a sketch map of the place. 

In other aspect of the present invention, there is provided a multi-level object data 
1 0 structure in a system for displaying multimedia information, wherein supplementary information 
on each object included in the multimedia information has multiple supplementary information 
structure including at least content information and real information on a particular program. 

In another aspect of the present invention, there is provided a multi-level object data 
structure wherein contents of a moving picture is expressed in a graph of object and place 
15 relation and an array of related events are made for each relation for making relation, and, for 
providing a function for displaying a required section, the moving picture is expressed in an 
object structure for providing information on objects, a place structure for providing information 
on a place which is a background of an event, an event structure for providing information 
required for describing a particular unit event in the moving picture, and relations of the 
20 structures in a graph, for searching the moving picture by using a double structure of content 
information and real information on at least one relation for each of the relations or a selected 
element, in the moving picture. 

The object structure, the place structure, and the event structure, expressed as content 



information and real information of the moving picture, are expressed in text information 
describing at least object name, place name, and event. 

The content information on the moving picture is information on a place or an object 
having a meaning in view of content of the moving picture, and the real information is 
5 information on a real place which is meaningful place in view of the content of the moving 
picture, or on actual performer cast an object in the moving picture. 

In further aspect of the present invention, there is provided a system for browsing a 
moving picture by using a multi-level object data structure including means for supplying 
moving picture digital information including a multiple object information structure having 
1 0 content based meaningful object information of the moving picture and real object information 
of the object, means for receiving the digital information and separating and reading the multiple 
object information and the moving picture information, and means for presenting real object 
information the multiple object information describes in response to a users request. 

The multiple object information structure has multiple object information structures for 
1 5 the obj ect, the place, and the event, and the means for presenting real obj ect information presents 
real information on an object cast in the moving picture or real place information on the place 
which is a background of an event. 

It is to be understood that both the foregoing general description and the following 
detailed description are exemplary and explanatory and are intended to provide further 
20 explanation of the invention as claimed. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The accompanying drawings, which are included to provide a further understanding of 
the invention and are incorporated in and constitute a part of this specification, illustrate 



embodiments of the invention and together with the description serve to explain the principles 
of the invention: 

In the drawings: 

FIG. 1 illustrates one frame of a moving picture for explaining the present invention; 

FIG. 2 illustrates a state of browsing of place information on a particular object when a 
moving picture is displayed in accordance with a preferred embodiment of the present invention; 

FIG. 3 illustrates a block diagram of a display for browsing a particular object in a 
moving picture in accordance with another preferred embodiment of the present invention; 

FIG. 4 illustrates a place DS of a particular object when a moving picture is displayed in 
accordance with a preferred embodiment of the present invention; 

FIG. 5 illustrates a semantic DS showing relation with object/place included in a moving 
picture in a form of a graph in searching/browsing the moving picture in accordance with a 
preferred embodiment of the present invention; 

FIG. 6 illustrates a detail of the event DS in FIG. 5; 

FIG. 7 illustrates a relation between object and place included in a moving picture in a 
form of object/place/event relation graph in displaying a moving picture in accordance with a 
preferred embodiment of the present invention; 

FIG. 8 illustrates multiple object DS included in a moving picture in accordance with a 
preferred embodiment of the present invention; 

FIG. 9 illustrates an character DS in displaying a moving picture in accordance with a 
preferred embodiment of the present invention; and, 

FIG. 10 illustrates a state of browsing of a particular object included in amoving picture 
by using multiple object DS in accordance with a preferred embodiment of the present invention. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
Reference will now be made in detail to the preferred embodiments of the present 
invention, examples of which are illustrated in the accompanying drawings. A transmission side, 
such as a VOD system or a digital TV broadcasting station transmits a particular video signal to 
5 a display in a receiver side according to a request from a user. In this instance, the video signal 
is transmitted, together with coded multiple information having detailed supplementary 
information on each object included in the video signal. Therefore, the display in the receiver 
side receives all the video signal and the detailed supplementary information from the 
transmission side, and displays the video signal only. FIG. 1 shows one scene of a drama with 
1 0 a hospital in a background. During watching a video program a receiver side display displays, 
the watcher may request for browsing detailed supplementary information on an object, such as 
the hospital by handling a controller, such as a remote controller. The detailed supplementary 
information may be program content based information of which video signal is provided, or 
actual information. If the watcher in the receiver side requests for browsing of actual information 
1 5 on the hospital shown in FIG. 1 , the receiver side browses and displays a name 201 , an address 
202, and description of the hospital 203, and a sketch map 204 of the actual hospital as shown 
in FIG. 2. On the other hand, the watcher may request for browsing of content information on 
the hospital in the picture in FIG. 1 for better understanding of the story of the video program. 
In this instance, the receiver side display may browse and display the detailed information on the 
20 content basis hospital according to such a users request. In FIG. 2, if a network environment is 
supported for the receiver side display such that the receiver side display can be connected to the 
internet, the present frame on the display may be linked, and shift to an internet address(URL) 
where there is more detailed information on the place. In FIGS. 1 and 2, though the user requests 



for browsing content based information or actual information on a place, like the hospital, by 
handling the remote controller in the middle of watching the video program, the same method 
may be applicable to an object or an event. That is, actual information on a content based 
specific object in the moving picture program may be requested. The object may be extended 
5 even to an animal or virtual character in the moving picture program. For example, necessary 
detailed supplementary information on a performer the user likes, among performers in a drama, 
such as information on a name, profile, a fan club, an internet home page address, and the like, 
may be requested. 

FIG. 3 illustrates a block diagram of a display for browsing a particular object in a 
1 0 moving picture in accordance with another preferred embodiment of the present invention. 

That is, FIG. 3 is an example for serving actual object related information as a coded data 
based on multiple information structure other than the moving picture, wherein the VOD server 
301 and the internet server 302 shown in FIG. 3 are connected to a communication network 309, 
and the communication network 309 and a digital TV broadcasting system 303 are connected to 
1 5 a receiver 305 of the receiver side display 304. In FIG. 3, the receiver 305 in the receiver side 
display 304 receives a video signal from the communication network 309 or the broadcasting 
system 303, and a decoder 306 decodes the video signal, to separate an original moving picture 
data and a supplementary information data. Then, the separated original video data is displayed 
on the display directly, while the supplementary information data is stored in a memory 307, for 
20 using on request for browsing of content based supplementary information or actual 
supplementary information on a particular object in the moving picture. In FIG. 3, the control 
unit 308 controls browsing operation for the detailed supplementary information of the object 
in response to the request of the watcher. In this instance, if the receiver side display 304 is 
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supported such that the receiver side display 304 is interlocked with a VOD system server or an 
internet server, the supplementary information of each object included in the video signal is not 
encoded with an original moving picture data, but is managed at the VOD server or the internet 
server separately, and provided at the watchers request. In the present invention, there is no 
special limitation on a position and a searching method of each object included in the video 
signal, and when and how the browsing is conducted. Therefore, as explained, the method for 
displaying the detailed supplementary information of each obj ect included in the video signal will 
bring about a new advertising effect. For example, if a particular cafe, restaurant, or the like 
provides the detailed supplementary information on the cafe, restaurant, or the like as the cafe, 
restaurant, or the like support the place in production of the moving picture program, the watcher 
can obtain supplementary information(for example, place, address, description, and sketch map 
and etc.,) on the actual supported place in the moving picture when necessary. 

FIG. 4 illustrates a place DS of a particular object when a moving picture is displayed in 
accordance with a preferred embodiment of the present invention. Referring to FIG. 4, the place 
DS 401 of the present invention includes a real location 402 and a content location in a moving 
picture. The content location 403 includes an annotation 404, a description of a name, or 
location of the place in a form of a text. And, the real location 402 includes a geometrical info 
405 and an annotation 406, a description of a name, or location of the real place in a form of a 
text. Therefore, the content location information is used for searching a moving picture by using 
object/place relation graph, and the real place information is used in providing information on 
the real place at users request. 

FIG. 5 illustrates a semantic DS showing relation with object/place included in a moving 
picture in a form of a graph in searching/browsing the moving picture in accordance with a 

10 



preferred embodiment of the present invention. Referring to FIG. 5, the semantic DS 501 of the 
present invention includes an event DS 502, an object DS 503, a place DS 504, an 
event/object/place relation DS 505 in which event /object/place relation are shown in a graph. 
The object DS, the place DS, and the event DS includes a text information for explaining the 

5 object, the place, and the event, respectively. 

FIG. 6 illustrates a detail of the event DS in FIG. 5. Referring to FIG. 6, the event DS 
601 of the present invention includes a reference to segment 602 and an annotation DS. 
Therefore, the receiver side display can display a relevant moving picture section by using such 
an event DS. That is, while text information explaining an event is provided, a moving picture 

1 0 section corresponding to the event is displayed by using the segment information. And, in FIG. 
6, one event may includes lower level event information explained in detail relatively. By using 
the event DS of such a hierarchical structure, object/place relation graph can be applied, not to 
an entire section of the moving picture, but to a portion of the section. 

FIG. 7 illustrates a relation between object and place included in a moving picture in a 

1 5 form of object/place/event relation graph in displaying a moving picture in accordance with a 
preferred embodiment of the present invention. Referring to FIG. 7, the object/place/event 
relation DS 701 of the present invention may be represented in an entity relation 702, inclusive 
of a reference to place 703, a reference to object 704, and a reference to event 705. In the object 
/place relation, one or more events may be related to one another, or only object and event are 

20 related, or place and event are related. 

FIG. 8 illustrates multiple object DS included in a moving picture in accordance with a 
preferred embodiment of the present invention. Referring to FIG. 8, the multiple object DS 801 
of the present invention includes a real object 802 and a content object 803 in a moving picture. 



The content object 803 includes an annotation which explains an object name or other 
information in a form of text, and the real object 802 includes an annotation 805 which explains 
the real object in a form of text. Thus, the content object is used for understanding a content of 
the moving picture, such as a moving picture searching by using an object/place(event when 

5 extended) relation graph, and the real object information is used for browsing or searching for 
the real object at a users request. 

FIG. 9 illustrates a multiple object DS in FIG. 8 in accordance with another preferred 
embodiment of the present invention, showing a multiple character DS. Referring to FIG. 9, the 
character DS 901 of the present invention includes a real character 902 and a content character 

10 903 . The content character 903 includes an annotation 904 which explains character information 
created according to name, character, and story development of the object in a content of the 
moving picture in a form of a text, and the real character 902 includes actor info of an actual 
particular object, and an annotation 906 explaining a name, a profile, the most important work 
and the like of the real character in a form of a text. The content character information is used 

15 for understanding a content of the moving picture, such as moving picture searching by using 
object/place relation graph, and real character is used in browsing/searching a real character 
(actor/actress) at users request. 

FIG. 10 illustrates a state of browsing of a particular object included in a moving picture 
by using multiple obj ect DS in accordance with a preferred embodiment of the present invention. 

20 In FIG. 10,101 expresses an outline of a video story by using the most schematic event structure, 
and 102 represents object-place relations set up within an event section selected in the outline 
101. And, 103 explains detailed events meeting relations(human B-hospital) selected on the 
object-place relation graph 102. 



Though multiple object DS of multiple character DS or multiple place DS is explained 
in the aforementioned embodiment of the present invention, multiple even DS can also be 
explained in the same method. For example, when contents of a moving picture is composed 
based on an actual event(or character), event information in a moving picture content and an 
actual event information may be provided on the same time. 

As has been explained, because each object included in a video signal has content basis 
supplementary information on a moving picture and actual supplementary information on the 
same time when a moving picture signal is displayed on a display, the watcher may use the 
supplementary information in browsing/searching the moving picture for understanding content 
of the moving picture when necessary, as well as the actual supplementary information, thereby 
providing a new concept of advertising. 

And, by providing object, place, and event structures, and a semantic structure 
representing relation between the structures, a moving picture browsing/searching can be made 
effectively based on the three elements when the moving picture is displayed. And, in order to 
carry out effective display of multimedia data of the present invention, particular object data 
structure is provided. 

It will be apparent to those skilled in the art that various modifications and variations can 
be made in system, method, and multi-level object data structure thereof for browsing a multi- 
media data of the present invention without departing from the spirit or scope of the invention. 
Thus, it is intended that the present invention cover the modifications and variations of this 
invention provided they come within the scope of the appended claims and their equivalents. 
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What is Claimed is: 

1. A method for browsing a multimedia data using a multilevel object data structure, 
comprising the steps of: 

(1) receiving multimedia information and multiple supplementary information on each 
5 object in the multimedia information on the same time; 

(2) separating the multimedia information and the multiple supplementary information; 

(3) displaying the multimedia information; and, 

(4) browsing/searching supplementary information related to a particular object from the 
multiple supplementary information, and displaying the supplementary information when a user 

1 0 requests for browsing/searching the supplementary information related to the particular object. 

2. A method as claimed in claim 1 , wherein the object related supplementary information 
includes; 

content information on a particular program in the multimedia information, and 
multiple supplementary information including real information. 



15 3. A method as claimed in claim 2, wherein the content information and the real 

information includes selective combinations of information which represents relations of object 
information, event information, place information, and object/e vent/place information in a graph. 



4. A method as claimed in claim 3, wherein the content information and the real 
information includes; 
20 text information describing character of the object, and 

14 



image or graphic information on the character. 

5. A method as claimed in claim 3, wherein the object information includes; 
text information having information describing the object, and 

image information on the object information. 

6. A method as claimed in claim 3, wherein the event information includes; 
text information describing contents of the event, and 

information on a place or a sketch map the event occurred. 

7. A method as claimed in claim 3, wherein the place information includes; 
text information describing the place, and 

information on location or a sketch map of the place. 

8. A multi-level object data structure in a system for displaying multimedia information, 
wherein supplementary information on each object included in the multimedia information has 
multiple supplementary information structure including at least content information and real 
information on a particular program. 

9. A multi-level object data structure as claimed in claim 8, wherein the content 
information and the real information includes selective combinations of semantic information 
which represents relations of object information, event information, place information, and 
object/e vent/place information in a graph. 

15 



10. A multi-level object data structure as claimed in claim 9, wherein the content 
information and the real information includes; 

text information describing character of the object, and 
image or graphic information on the character. 

11. A multi-level object data structure as claimed in claim 9, wherein the object 
information includes; 

text information having information describing the object, and 
image information on the object information. 

12. A multi-level object data structure as claimed in claim 9, wherein the event 
information includes; 

text information describing contents of the event, and 
information on a place or a sketch map the event occurred. 

13. A multi-level object data structure as claimed in claim 9, wherein the place 
information includes; 

text information describing the place, and 
information on location or a sketch map of the place. 

14. A multi-level object data structure wherein contents of a moving picture is expressed 
in a graph of object and place relation and an array of related events are made for each relation 
for making relation, and, for providing a function for displaying a required section, the moving 

16 



picture is expressed in an object structure for providing information on objects, a place structure 
for providing information on a place which is a background of an event, an event structure for 
providing information required for describing a particular unit event in the moving picture, and 
relations of the structures in a graph, for searching the moving picture by using a double structure 
5 of content information and real information on at least one relation for each of the relations or 
a selected element, in the moving picture. 

15. A multi-level object data structure as claimed in claim 14, wherein the object 
structure, the place structure, and the event structure, expressed as content information and real 
information of the moving picture, are expressed in text information describing at least object 

1 0 name, place name, and event. 

16. A multi-level object data structure as claimed in claim 14, wherein the content 
information on the moving picture is information on a place or an object having a meaning in 
view of content of the moving picture, and the real information is information on a real place 
which is meaningful place in view of the content of the moving picture, or on actual performer 

1 5 cast an object in the moving picture. 

17. A system for browsing a moving picture by using a multi-level object data structure 
comprising: 

means for supplying moving picture digital information including a multiple object 
information structure having content based meaningful object information of the moving picture 
20 and real object information of the object; 

17 



means for receiving the digital information and separating and reading the multiple object 
information and the moving picture information; and, 

means for presenting real object information the multiple object information describes 
in response to a users request. 

18. A system as claimed in claim 17, wherein the multiple object information structure 
has multiple object information structures for the object, the place, and the event, and the means 
for presenting real object information presents real information on an object cast in the moving 
picture or real place information on the place which is a background of an event. 
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ABSTRACT 

System, method, and multi-level obj ect data structure thereof for browsing a multi-media 
data, the method for browsing a multimedia data using a multilevel object data structure 
including the steps of (1) receiving multimedia information and multiple supplementary 

5 information on each object in the multimedia information on the same time, (2) separating the 
multimedia information and the multiple supplementary information, (3) displaying the 
multimedia information, and (4) browsing/searching supplementary information related to a 
particular object from the multiple supplementary information, and displaying the supplementary 
information when a user requests for browsing/searching the supplementary information related 

1 0 to the particular obj ect. 
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DECLARATION AMD POWER OF ATTORNEY 



As a below named inventor, I hereby declare that: 

My residence, post office and citizenship are as stated below next to my name, 

I believe I am the original, first and sole inventor (if only one name is listed below) or an original, first and joint inventor (if plural names are listed below) of the subject matter 
claimed and for which a patent is sought on the invention entitled 

SYSTEM, METHOD. AND MULTI-LEVEL OBJECT DATA STRUCTURE THEREOF FOR BROWSING MULTIMEDIA DATA 



, the specification of which 

[ X ] is attached hereto [ ] was filed on as Application Serial No and was 

amended on (if applicable) 

I hereby state that I have reviewed and understand the contents of the above identified specification, including the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose information which is known to me to be material to patentability in accordance with Title 37, Code of Federal Regulations, Section 1 .56(a). 

I hereby claim foreign priority or provisional application benefits under Title 35, United States Code, Section 1 19 of any foreign application(s) for patent or inventor's 
certificate, or provisional application(s) listed below and have also identified below any foreign application for patent or inventor's certificate, or provisional application(s) 
having a filing date before that of the application on which priority is claimed: 

Prior Foreign Application(s) or U.S. Provisional Application^): Priority Claimed 

Isumber Country Day/Month/Y ear Yes No 

2&&3/1999 Korea 3 / July/1999 X 



f=2 I hereby claim the benefit under Title 35, United States Code, Section 120 of any United States apphcation(s) listed below and, insofar as the subject matter of each 
oI!|e claims of this application is not disclosed in the prior United States application in the manner provided by the first paragraph of Title 35, United States Code, Section 
H#, 1 acknowledge the duty to disclose material information as defined in Title 37, Code of Federal Regulations, Section 1.56(a) which occurred between the filing date of 
the prior application and the national or PCT international filing date of this application: 

Prior U. S. Application(s): 

Serial No. Filing Date Status: Patented. Pending, Abandoned 



I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information and belief are believed to be true; and further that 
these statements were made with the knowledge that willful false statements and the like so made are punishable by fine or imprisonment, or both, under Section iOOl of 
Title 18 of the United States Code and that such willful false statements may jeopardize the validity of the application or any patent issued thereon. 



I hereby appoint the following attomey(s) and/or a S ent(s):Daniel Y.J. Kim, Registration No. 36,186 and Mark L. Fleshner, Registration No. 34,596; Carl R. Wesolowski, 
Registration No. 40,372, John C. Eisenhart, Registration No. 38,128, Rene A. Vasquez, Registration No. 36,647, all of 



The Law Offices of 
FLESHNER & KIM 
P. O. Box 221200 
Chantilly, Virginia 20153-1200 

With full power of substitution and revocation, to prosecute this application and to transact all business in the Patent and Trademark Office connected therewith, and all future 
correspondence should he addressed to them. 
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of sole or first inventor: Jin Sftp LEE 

Date: May 9,2000 



Residence: Seoul, Korea 

Citizenship: Republic of Korea 

PosSbffice Address: Samho APT., 101-809, 136, Koyol-dong, Songpa-gu, Seoul, Korea 



tor(s): Hyeo 



Iriirentor's signature: 



Residence: Kyonggi-do, Korea 
(jjjzenship :Republic of Korea 



st Office Address: Hanshinlife, 1 09-302, Pundang-dong, Pundang-gu, Songnam-shi, Kyonggi-do, Korea 



Citizenship: . 

Post Office Address: 
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